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/. Real Party in Interest 

The real party in interest in the present appeal is Sharp Kabushiki Kaisha, the 
assignee of the present application. 

//. Related Appeals and Interferences 

Appellants, appellants' legal representatives, and/or the assignee of the present 
application are unaware of any appeals or interferences which will directly affect, or be 
directly affected by or have a bearing on the Board's decision in the pending appeal. 

///. Status of Claims 

Claims 1-42 are pending in the application. Claims 1-21 and 35-38 have been 
allowed, and claims 23 and 31 have been indicated as being allowable if rewritten in 
independent form. Claims 22, 24-30, 32-34 and 39-42 stand finally rejected and are 
the subject of this appeal. 

IV. Status of Amendments 

On October 26, 2005, Applicant filed an amendment after final amending claim 
23. For purposes of appeal, the amendment was entered. 

V. Summary of Claimed Subject Matter 
A. Background 

To create a stereoscopic display, two images are acquired using a stereoscopic 
image capture device that provides two image capture devices. One image capture 
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device (known as the "left image capture device") captures an image corresponding to 
the image that would be seen by the left eye of an observer, and the other image 
capture device (known as the "right image capture device") captures an image 
corresponding to the image that would be seen by the right eye of an observer. The 
two images thus acquired are known as a pair of stereoscopic images, or stereoscopic 
image pair. When the two images are displayed using a suitable stereoscopic display 
device, a viewer perceives a three-dimensional image. 

One problem with conventional stereoscopic displays is that stereoscopic images 
can be uncomfortable to view, even on high quality stereoscopic display devices. One 
cause of discomfort is the presence of vertical disparity within a stereoscopic image 
pair. Vertical disparity means that the image of an object in one of the stereoscopic 
images has a different vertical position than the image of the same object in the other 
stereoscopic image. Vertical disparity arises due to many kinds of mis-alignment of the 
camera systems, and causes discomfort to a viewer. Image rectification is a process 
for eliminating vertical disparity between the two images of a stereoscopic image pair, 
thereby making the resultant stereoscopic image more comfortable to view. 

B. Description 

Referring to Fig. 6 (reproduced below), a method of rectifying a stereoscopic 
image that includes first and second captured images obtained from first and second 
image capture devices is shown. Fig. 6, portions of which correspond to claims 22 and 
39, illustrates an entire rectification process from initial capture of the image pair to 
display of the rectified image on a suitable stereoscopic imaging device. 

The method is intended for use with a pair of captured images that form a 
stereoscopic image pair, and this pair of images forms one input to the method. A 
suitable image capture device, for example, is a stereo-camera including of a pair of 
digital cameras. 
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At step 1 1 , a stereoscopic 
image pair consisting of a left image 
and a right image pair is captured 
with a stereoscopic camera set-up. 
At step 12, pixel correspondences 
between the left and right images 
are detected using any standard 
technique. 1 That is, pairs of image 
points are established (one point in 
the left image and one point in the 
right image) that are images of a 
unique three-dimensional point in 
the object scene (i.e., each pair 
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FIG 5 



a corresponding point in the other 
image). If there is vertical disparity 
between the two images, this will 
become apparent during correspondence detection. The correspondence of the point 
features is established using known robust statistical methods, which reject chance 
correspondences that do no fit into the epipolar geometry governed by the majority of 
the correspondences. 2 At step 18 these correspondences are used to compute a 
fundamental 1 matrix relating the two images. 3 



1. Page 29. line 20-page 30, line 5 of the application. Steps 11, 12, 18 and 19 of Fig. 6 are 
identical to the respective steps of Fig. 5. The specification does not explicitly discuss these steps with 
respect to Fig. 6 and, thus, reference is made here to the description relating to Fig. 5. 

2. Page 48, lines 12-21 of the application. 

3. Page 30. lines6-7 of the application. 
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The fundamental matrix relates a point Xq in the left image of an image pair to 
the corresponding point x, in the right image of the image pair. For example, the 
camera matrices can be given by 



P x = tf l |* r |-* r 'J 

where K, is a 3x3 calibration matrix of the /, h camera (/ = 0 for the left camera and 
/ = 1 for the right camera), R is a 3x3 rotation matrix and t is a translation 3-vector. R 
and t are respectively the rotation and translation of the right camera (/ - 1) relative to 
the left camera (/= 0). Assuming that skew is negligible, the calibration matrix K, \s 



'/, 0 Pi 
0 /, g, 
0 0 0 



where f, is the effective focal length and [p>q) is the principal point in the image plane. 
The 3x3 fundamental matrix F relates to the projective and similarity components of the 
rectifying transformation, as indicated by step 18. 4 

As recited in claims 22 and 39, first and second rectification transformations are 
determined and are used for rectifying a respective one of the first and second images 
(obtained in step 1 1) so as to reduce vertical disparity. The first and second 
rectification transforms are determined so that they correspond to a virtual alignment to 
a parallel camera set-up. Moreover, claims 22 and 39 further recite that the 
determination of the first and second rectification transformations includes calculating^. 



4. Page 46. line 12-page 47, line 5 of the application. 
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shear component such that a final matrix is a combination of a rotation end a translation 
and at least one camera parameter These features are discussed in more detail 
below. 

At step 19, the correspondence information is used to determine a component of 
the rectification transformations (the "projective and similarity components") which will 
be used to rectify the two images. This component of the overall rectification 
transformations is intended to remove vertical disparity from the rectified image pair. 5 

In the embodiment of Fig. 6 (portions of which correspond to claims 22 and 39), 
it is assumed that the focal length and principal points of the left and right cameras are 
known (e.g., from tests made during manufacture) and this data is input at step 25. At 
step 26, the rotation and translation operators R and t are estimated from the focal 
length and principal points of the cameras, and from the projective and similarity 
components of the transformations. This is done by decomposing the final matrix to be 
calculated into several parts, most of which are known. Standard mathematical 
methods are then used to solve for the unknown quantities. 6 

Once the rotation and translation operators have been estimated, then at step 22 
the horizontal shear and scaling components of the rectification transformation are 
determined from the known focal lengths and principal points of the cameras, and from 
the estimated rotation and translation operations. The pair of rectification 
transformations are then found by combining the projective and similarity component of 
the transformations (step 19) with the horizontal shear and scale components (step 22 
with data from steps 25 and 26). If desired, the estimated camera rotation and 
translation operations can be output at step 27. 



5. Page 30, lines 1 1-16 of the application. 

6. Page 36, line 18-page 37. line 2 of the application. 
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The invention of claims 22 and 39 is particularly suited for processing a stereo 
image pair captured using a stereo camera set-up where the intrinsic camera 
parameters are accurately known, but the extrinsic parameters are not accurately 
known - that is, where each camera is individually of high quality, and the deviation of 
the stereoscopic camera set-up from a parallel camera set-up occurs primarily in the 
orientation of one camera relative to the other. 

In the invention of claims 22 and 39, the choice of the horizontal shear and 
scaling components of the transformations is constrained to ensure that the resultant 
pair of rectifying transformations corresponds to a virtual alignment to a parallel camera 
set-up. To ensure this, the shear component is calculated from an equation formulated 
such that the final matrix is a combination of a rotation and a translation and the internal 
camera parameters, 7 as recited in claims 22 and 39. 

Criteria used to determine a shear component in prior art methods can lead to 
rectification transformations that do not correspond to a virtual alignment to a parallel 
camera setup. According to the invention of claims 22 and 39, the criterion for the 
determination of the shear component relates to what is physically probable. The shear 
component is chosen such that the rectifying homography corresponds to virtually 
rotating the camera. Furthermore, the shear terms are constrained using a priori 
knowledge of the intrinsic and extrinsic parameters of the camera. This a priori 
knowledge is expressed in terms of probability densities. 8 

After the projective and similarity component of the transformation and the 
horizontal shear and scaling component of the transformation have been determined, 
they are combined at step 23 to produce the pair of rectifying transformations at step 
14. Once determined, the rectification transformations may be used immediately, or 



7. Page 37, line 4-page 38. line 12 of the application. 

8. Page 56, line 21 -page 57, line 15 of the application. 
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they may be output and/or stored for subsequent use. When the rectification 
transformations are used, they are used to warp the captured image pair in a 
conventional manner at step 15, to produce a rectified image pair at step 16. The end 
product is a rectified image pair, with no or substantially no vertical disparity, which 
should be much more suitable for comfortable stereoscopic viewing than the original 
captured image pair. 

The rectified image pair may be displayed on a suitable stereoscopic display 
device at step 17, for direct viewing by an observer. Alternatively, the rectified image 
pair can be stored for future use. 

VI. Grounds of Rejection to be Reviewed on Appeal 

A. Whether claims 22, 24-27, 33-34, and 39-40 are patentable under 35 
U.S.C. §103(a) over IEEE article by Courtney et al. (A Hardware 
Architecture for Image Rectification and Ground Plane Obstacle 
Detection, IEEE, 1992, hereinafter Courtney) in view of U.S. Patent No. 
6,608,923 to Zhang et al. (hereinafter Zhang). 

B. Whether claims 28-30, 32 and 41-42 are patentable under 35 U.S.C. 

§1 03(a) over Courtney in view of Zhang in further view of U.S. Patent No. 
5,142,357 to Lipton et al. (hereinafter Upton) and Applicant's admitted 
prior art (AAPA). 

VII. Argument 

Claims 22, 24-30, 32-34 and 39-42 stand finally rejected as being unpatentable 
over the applied art. For the following reasons, it is respectfully submitted that the 
claims are patentable over the applied art and the final rejection should be withdrawn. 

A. Claim 24 

Claim 24 stands rejected rejected under 35 USC §1 03(a) as being unpatentable 
over Courtney in view of Zhang. Reversal of the rejection is respectfully requested for 
at least the following reasons. 
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In the Advisory Action mailed November 15. 2005, claim 23 was indicated as 
being allowable if rewritten in independent form including all the limitations of the base 
claim and any intervening claims. Since claim 24 depends from claim 23. claim 24 is 
allowable for at least the same reasons. 

Accordingly, reversal of the rejection of claim 24 is respectfully requested. 
B. Claims 22, 25-30, 32-34 and 39-42 
1. Claims 22, 25-27, 33-34 and 39-40 

Claims 22. 25-27, 33-34, and 39-40 stand rejected under 35 USC §1 03(a) as 
being unpatentable over Courtney in view of Zhang. Reversal of the rejection is 
respectfully requested for at least the following reasons. 

Independent claims 22 and 39 recite that a shear component is calculated such 
that a final matrix is a combination of a rotation and a translation and at least one 
internal camera parameter. The Examiner admits that Courtney does not teach this 
feature. However, the Examiner contends that Zhang does teach this feature, and cites 
to the Abstract, column 6. lines 55-65 and Fig. 7 of Zhang} 

Zhang describes a method for rectification of images so as to achieve minimal 
distortion, wherein images are rotated and translated, and then a shearing transform is 
applied to scale and translate the images in the horizontal direction. 10 Zhang, however, 
has not been found to teach or suggest calculating a shear component such that a final 



9. Page 3, first full paragraph of the Office Action 

10. Fig. 7 of Zhang. 
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matrix is a combination of a rotation and a translation and at least one internal camera 
parameter, as recited in claims 22 and 39. 

Applicants respectfully submit that Zhang does not make up for the deficiencies 
in Courtney. As will be appreciated in view of the following, Applicants respectfully 
submit that Zhang does not teach or suggest determining the first and second 
rectification transformation so that the first and second rectification transformations 
correspond to a virtual alignment to a parallel camera set-up. wherein determining the 
firet and second rectification transformation includes calculating a shear component 
such that a final matrix is a combination of a rotation and a translation and at least one 
internal camera parameter. Thus, even if the teachings of the references were 
combined as proposed by the Examiner, the invention of claims 22 and 39 would not 
result. 



.700 



Referring now to the cited 
portions of Zhang, the Abstract 
simply states that affine 
transforms, which are used to 
compute two-dimensional 
projective transforms or 
homographies, can include a 
first transform and an optional 
second transform for image 
rectification. Fig. 7 of Zhang 
(reproduced at right) illustrates 
the steps for providing 
rectification of images from a 
projective transform and 
fundamental matrix. The affine 



SPECIAL PROJECTIVE TRANSFORM 
& FUNDAMENTAL MATRIX 



y71€ 



FIND AFFINE COMPONENT OF EACH HOLOGRAPHY 



SIMILARITY TRANSFORM 



ROTATE AND TRANSLATE IMAGES SUCH THAT THE EPFOLAR 
UNES ARE HORIZONTALLY ALIGNED SO THAT IT SATIRES THE 
CONSTRAINTS FOR RECTIFICATION 



SHEARING TRANSFORM 



SCALE AND TRANSLATE IMAGES IN HORIZONTAL DIRECTION 
WHILE PRESERVING 0R3ONAL IMAGE RESOLUTION AS CLOSE 
AS POSSIBLE FOR FURTHER REDUCE DISTORTION 
INTRODUCED BY THE PROJECTIVE COMPONENT 



I 



,712 



,719 



RECTIFICATION OF IMAGES WITH MINIMUM 
DISTORTION 




720 



FIG. 7 



PAGE 10/19 1 RCVD AT 2/8/2006 8:33:56 AM [Eastern Standard Time] 1 SVR:USPTOEFXRF-6/26 * DNIS:2738300 • CSID:216 621 6165 * DURATION (mm-ss):05-32 



02/08/06 08:40 FAX 216 621 6165 



RENNER OTTO 2 0 



-» USPTO GENERAL 0011 



Serial No.:10/081,916 
Page 1 1 

component of each homography is obtained by performing a similarity transform (step 
712), wherein the affine component is based on images that are rotated and translated 
such that epipolar lines are horizontally aligned to satisfy the constraints for rectification. 
Then, a shearing transform is applied (step 714), wherein images are scaled and 
translated in the horizontal direction while attempting to preserve the original image 
resolution (step 718). Neither the Abstract nor Fig. 7, however, have been found to 
teach or suggest that a shear component is calculated such that a final matrix is a 
combination of a rotation and a translation and at least one internal camera parameter. 
as recited in claims 22 and 39. 

Further, column 6, lines 55-65 of Zhang state that if intrinsic parameters of a 
camera are known, the images are calibrated, and the fundamental matrix becomes the 
essential matrix. More importantly, however, the cited portion states that the invention 
is applicable to both calibrated or uncalibrated image pairs. In other words, the 
invention disclosed in Zhang is applicable to systems wherein the intrinsic camera 
parameters are known (calibrated images) or unknown (uncalibrated images). 
Moreover, intrinsic camera parameters do not appear in any of the formulae of Zhang. 
Thus, it follows that the method of Zhang can be applied without knowledge of intrinsic 
camera parameters (e.g., uncalibrated images). 

Additionally, Zhang has not been found to teach or suggest calculating the shear 
component such that a final matrix is a combination of a rotation and a translation and 
at least one internal camera parameter, as recited in claims 22 and 39. This is logical, 
as the method of Zhang is applicable to uncalibrated images (images obtained without 
knowledge of intrinsic camera parameters) as well as calibrated images. Thus, intrinsic 
camera parameters are not required to implement the method of Zhang. Zhang also 
has not been found to teach or suggest how one would use intrinsic camera parameters 
to determine the transformations, nor are they shown with respect to the shearing 
transform. 
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Accordingly, Courtney in view of Zhang has not been found to teach or suggest 
that a *h*ar component is calculated such that a final matrix is a combination of a 
rotation and a translation and at least one internal camera parameter, as recited in 
claims 22 and 39. 

Further, in the Advisory Action dated November 15, 2005, the Examiner, with the 
exception of claim 23, maintains the rejections of the final Office Action. Additionally, 
the Examiner states that Courtney discloses using internal camera parameters for 
computing matrix elements, and that Zhang teaches that a fundamental matrix is 
rotated and translated, such that it satisfies the constraints for rectification. 11 According 
to the Examiner, it would have been obvious to combine Courtney and Zhang to arrive 
at the invention of claim 22. 

While section 2 of Courtney does discuss the use of camera parameters, 
Courtney has not been found to teach or suggest that a shear component is calculated 
such that a final matrix is a combination of a rotation and a translation and at least one 
internal camera parameter. Instead, Courtney simply states that matrix elements may 
be computed from camera calibration parameters. There is no discussion in section 2 
of Courtney with respect to calculating a shear component 

Moreover, and as discussed above, none of the formulae in Zhang include 
camera parameters. Further, while Zhang does discuss intrinsic camera parameters, 
the disclosure of Zhang indicates that the method can be implemented with or without 
knowledge of the camera parameters. 12 Thus, one viewing Zhang would have no 
motivation to combine the camera parameters of Courtney (which do not relate to a 



1 1 . Section 1 1 of the Advisory Action dated November 1 5, 2005. 

1 2, See column 6. lines 55-65 of Zhang and argument on pages 11-12. 



PAGE 12/19 * RCVD AT 2/812006 8:33:56 AM [Eastern Standard Time] 1 SVR:USPTO-EFXRF-6/26 * DNIS:2738300 1 CSID:216 621 6165 * DURATION (mm-ss):05-32 



02/08/06 08:40 FAX 218 621 6165 



RENNER OTTO 2 0 



-» USPTO GENERAL ©013 



Serial No.:10/081,916 
Page 13 

shear component), with the method of Zhang (which does not need camera 
parameters). 

Accordingly, reversal of the rejection of claims 22 and 39 is respectfully 
requested. 

Claims 25-27, 33-34, and 40 directly or indirectly depend from claim 22 or 39 or 
claim 39 and, therefore, can be distinguished from the cited art for at least the same 
reasons. 

Accordingly, reversal of the rejection of claims 25-27, 33-34, and 40 is 
respectfully requested. 

Z Claims 28-30, 32 and 41-42 

Claims 28-30, 32 and 41-42 are rejected under 35 USC § 103(a) as being 
unpatentable over Courtney and Zhang in further view of U.S. Patent No. 5,142,357 to 
Upton et al. (hereinafter Upton) and Applicant's admitted prior art (AAPA). Reversal of 
the rejection is respectfully requested for at least the following reasons. 

Claims 28-30 and 32 directly or indirectly depend from independent claim 22, 
while claims 41-42 directly or indirectly depend from independent claim 39. As 
discussed above, claims 22 and 39 can be distinguished from Courtney and Zhang. 

Upton is cited for teaching the use of a data processor and that a stereoscopic 
camera can be used for video or still images. 13 Neither Upton nor the AAPA, however, 
have been found to make up for the above-noted deficiencies of Courtney and Zhang. 



1 3. Page 5. last two paragraphs of the Office Action dated August 8, 2005. 
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Thus, claims 22 and 39 can be distinguished from Courtney, Zhang, Upton and the 
AAPA. 

Since claims 28-30, 32 and 41-42 depend from one of the above independent 
claims, they can be distinguished from the cited art for at least the same reasons. 

Accordingly, reversal of the rejection of claims 28-30, 32 and 41^2 is 
respectfully requested. 

Conclusion 

In view of the foregoing, appellant respectfully submits that the claims are 
patentable over the applied art and that the final rejection should be reversed. 

Respectfully submitted, 

RENNER, OTTO, BOISSELLE & SKLAR 




Kenneth W. Fafrak 
Registration No. 50,689 
1621 Euclid Avenue 
Cleveland, Ohio 44115 
(216) 621-1113 
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VIII. Claims Appendix 



22. A method of rectifying a stereoscopic image comprising first and second 
images captured using first and second image capture devices, the first and second 
image capture devices forming a stereoscopic image capture device, the method 
comprising the step of; 

determining first and second rectification transformations for rectifying a 
respective one of the first and second images so as to reduce vertical disparity; 
characterised in that the method comprises determining the first and second 
rectification transformation so that the first and second rectification transformations 
correspond to a virtual alignment to a parallel camera set-up, wherein determining the 
first and second rectification transformation includes calculating a shear component 
such that a final matrix is a combination of a rotation and a translation and at least one 
internal camera parameter. 

24. A method as claimed in claim 23 wherein the statistics of the parameters 
of the stereoscopic image capture device relate to the alignment of the first image 
capture device relative to the second image capture device. 

25. A method as claimed in claim 22 wherein the step of determining the first 
and second rectification transformations comprises: 

determining a first component of each of the first and second rectification 
transformations, the first component of the first rectification transformation and the first 
component of the second rectification transformation substantially eliminating vertical 
disparity from the rectified image pair; and 

determining a second component of each of the first and second rectification 
transformations so that the first and second rectification transformations correspond to 
a virtual alignment to a parallel camera set-up. 



PAGE 15/19 1 RCVD AT 2/8/2006 8:33:56 AM [Eastern Standard Time] 1 SVR:USPTO*EFXRF-6/26 * DNIS:2738300 1 CSID:21 6 621 6165 * DURATION (mm-ss):05-32 



•02/03^06 08:41 FAX 216 621 6165 



RENNER OT TO 20 



-> USPTO GENERAL @!016 



Serial No.:10/081,916 
Page 16 

26. A method as claimed in claim 25 and further comprising the step of using 
statistics of the parameters of the stereoscopic image capture device in the step of 
determining the first and second rectification transformations wherein the statistics of 
the parameters of the stereoscopic image capture device are used in the step of 
determining the second components of the first and second rectification 
transformations. 

27. A method as claimed in claim 25 wherein the statistics of the parameters 
of the stereoscopic image capture device relate to the alignment of the first image 
capture device relative to the second image capture device. 

28. A method as claimed in claim 22 wherein the first captured image and 
second captured image comprise a still stereoscopic image. 

29. A method as claimed in claim 22 wherein the first captured image and 
second captured image comprise a frame of a stereoscopic video image, 

30. A method as claimed in claim 29 and comprising the steps of: 
determining first and second rectification transformations for a first frame of the 

stereoscopic video image using a method as defined in claim 22; and 

rectifying subsequent frames of the stereoscopic video image using the first and 
second rectification transformations determined for the first frame of the stereoscopic 
video image. 

32. A method as claimed in claim 29 and comprising the steps of: 
determining first and second rectification transformations for each frame of the 
stereoscopic video image using a method as defined in claim 22; and 
rectifying each frame of the stereoscopic video image using the first and second 
rectification transformations determined for that frame. 
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33. A method as claimed in claim 22 and comprising the further step of 
rectifying the first and second captured images using a respective one of the first and 
second rectification transformations. 

34, A method as claimed in claim 33 and comprising the further step of 
displaying the first and second rectified images on a stereoscopic display device for 
viewing by an observer. 

39. An apparatus for rectifying a stereoscopic image comprising first and 
second images captured using first and second image capture devices, the first and 
second image capture devices forming a stereoscopic image capture device, the 
apparatus comprising: 

means for determining first and second rectification transformations for rectifying 
a respective one of the first and second images so as to reduce vertical disparity, the 
first and second rectification transformations corresponding to a virtual alignment to a 
parallel camera set-up, wherein determining the first and second rectification 
transformation includes calculating a shear component such that a final matrix is a 
combination of a rotation and a translation and at least one internal camera parameter. 

40. An apparatus as claimed in claim 39 and further comprising means for 
rectifying the first and second captured images using a respective one of the first and 
second rectification transformations. 

41 . An apparatus as claimed in claim 39 and comprising a programmable 
data processor 

42. A storage medium containing a program for the data processor of an 
apparatus as defined in claim 41 . 
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IX. Evidence Appendix 

None 
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Related Proceedings Appendix 

None 
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